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Abstract 


A human-in-the-loop simulation experiment was conducted at the NASA Langley 
Research Center’s (LaRC) Air Traffic Operations Lab (ATOL) in an effort to 
comprehensively validate tools and procedures intended to enable the Small Aircraft 
Transportation System, Higher Volume Operations (SATS HVO) concept of operations. The 
SATS HVO procedures were developed to increase the rate of operations at non-towered, 
non-radar airports in near all-weather conditions. A key element of the design is the 
establishment of a volume of airspace around designated airports where pilots accept 
responsibility for self-separation. Flights operating at these airports, are given approach 
sequencing information computed by a ground based automated system. The SATS HVO 
validation experiment was conducted in the ATOL during the spring of 2004 in order to 
determine if a pilot can safely and proficiently fly an airplane while performing SATS HVO 
procedures. Comparative measures of flight path error, perceived workload and situation 
awareness were obtained for two types of scenarios. Baseline scenarios were representative 
of today’s system utilizing procedure separation, where air traffic control grants one 
approach or departure clearance at a time. SATS HVO scenarios represented approaches 
and departure procedures as described in the SATS HVO concept of operations. Results 
from the experiment indicate that low time pilots were able to fly SATS HVO procedures 
and maintain self-separation as safely and proficiently as flying today’s procedures. 


I. Introduction 

T he SATS HVO concept of operations was developed to address the forecasted shortfall in capacity in the Air 
Traffic System. Its goal is to increase the utilization of non-towered, non-radar airports during periods of 
Instrument Meteorological Conditions (IMC) without a major negative impact to the Air Traffic Control (ATC) 
structure or workload. Fundamental principles of this concept such as the functional division of air traffic 
sequencing and separation tasks, a volume of controlled airspace around an airport, and ground based sequencing 
automation were part of a method of separation assurance described in [1]. The study included results of a batch 
simulation experiment of airport performance and provided a reference list of seminal works in the subject area. An 
evolved version of the concept was depicted in a later work [2] that added a modified notion of timed approaches to 
increase the number of concurrent Instrument Flight Rules (IFR) operations at similar airports. The SATS HVO 
Concept of Operations relies on a set of procedures, the establishment of a volume of airspace around designated 
airports where pilots accept responsibility for self-separation referred to as the Self Controlled Area (SCA) and a set 
of avionics that includes Automatic Dependent Surveillance-Broadcast (ADS-B) and air-ground datalink 
communications. A complete description of the procedures and required equipment can be found in [3, 4]. Flights 
operating in the SCA during IMC are given approach sequencing information computed by a ground based 
automated system referred as the Airport Management Module (AMM). Transition procedures between the SCA and 
controlled airspace can be found in [5]. 

The SATS HVO human in the loop validation experiment was conducted in the ATOL during the spring of 2004 
to determine if a pilot can safely and proficiently fly an airplane while performing SATS HVO procedures. The 
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experiment also sought to assess pilot workload and situation awareness using HVO procedures and tools. 
Comparative measures of flight path error, perceived workload and situation awareness were obtained for two types 
of scenarios. Baseline scenarios were representative of today’s system utilizing procedural separation, where air 
traffic control (ATC) grants one approach or departure clearance at a time. SATS HVO scenarios represented 
approaches and departure procedures as described in the SATS HVO concept of operations [3]. Preliminary results 
from this experiment were published in [6]. The study also included a Monte Carlo analysis of throughput and 
approach delays that compared current day operations with SATS HVO. The same research objectives and 
experiment design were shared by a flight experiment conducted in the summer 2004 whose results can be found in 
[10]. The validation process included other studies that looked at different aspects of the concept. For example, in 
[7] the flight technical error metrics collected during simulation and flight experiments were compared in an effort 
to validate the simulation tools used in the ATOL. More recent pilot in the loop experiments investigated priority 
landings and visual flight rules (VFR) operations in the SCA as well as the use of some of the experimental avionics 
designed for the project [8, 9]. 

This paper provides a detailed description of the experiment design and results associated with subjective 
assessments of workload and situation awareness, procedure conformance and responses to usability questionnaires. 

The organization of this paper is as follows: Section 1 provides a brief overview of the concept of operations. 
Section 2 introduces the experiment methodology, simulation platform, and the experiment design. Sections 3 and 4 
contain the analyses of the experiment results and concluding remarks. 

II. Overview of the HVO Concept of Operations 

Key to the SATS HVO concept is the use of a newly defined area of flight operations called the SCA, 
established around SATS designated airports, (i.e., non-towered, non-radar equipped airports) during periods of 
IMC, similar to present day Class D Airspace that surrounds towered airports. 

Within the SCA, pilots, using advanced airborne systems, would have the ability and responsibility to maintain 
separation between themselves and other similarly equipped airplanes. Aircraft operating in this airspace would need 
special avionics, e.g., automated dependent surveillance-broadcast (ADS-B), a two-way data link, and appropriate 
self-separation tools in order to participate. This concept would also require a new, ground-based automation 
system, the AMM, typically located at the airport that would provide appropriate sequencing information to the 
arriving aircraft. The AMM provides an arrival sequence and broadcasts the total number of arriving aircraft in the 
SCA. It does not, however, provide separation, altitude assignments, or sequence departures. 

This proposed operational concept emphasizes the integration with the current and planned near-term National 
Airspace System by minimizing additional Air Traffic Control (ATC) workload. A simulation study conducted 
jointly by NASA LaRC and the Federal Aviation Administration (FAA) Technical Center focused on transition 
procedures between the SCA and ATC managed airspace. The study measured controller’s workload and 
acceptability of SATS HVO operations [5]. Results from the experiment showed that no significant increase in 
controller’s workload was observed during SATS HVO that resulted in a four-fold increase in operations. 

The SATS HVO concept is a starting point or “template” for additional designs and analyses. No attempts have 
been made to optimize the size or shape of the proposed airspace. To date, the development focus has been on 
providing an operational concept that was safe, would enable more than one operation at a time, and would not 
require significant ground infrastructure costs or improvements. Global Positioning System (GPS) “T” instrument 
approach procedures were chosen as a basis for this concept, although other instrument approach procedures could 
be used as well. 

Aircraft arriving into the SATS designated airport are managed by ATC according to an IFR flight plan to a 
transition fix above the SCA. The transition fix is also the Initial Approach Fix (IAF) on a GPS-T instrument 
approach procedure. Prior to reaching the transition fix, pilots must request landing sequence information from the 
AMM. The AMM message includes the SCA entry procedure (vertical or lateral), relative sequence information 
(follow <callsign>), and Missed Approach Holding Fix (MAHF) assignment. If the SCA is full, then the AMM 
sends a “stand by” message. An SCA configuration such as the one shown in Fig. 1 allows a maximum of four 
aircraft on approach ant any given time. The limit is based on the number of holding altitudes at the two MAHFs. 

Pilots in the SCA initiate their approach once adequate spacing behind the lead aircraft has been met (determined 
through either a generic rule-based spacing procedure, i.e., safe for all combinations of aircraft performance, or by 
using an on-board self-spacing tool). For SATS HVO departures, pilots will file flight plans with a SATS HVO 
departure procedure to a Departure Fix (DF), obtain ATC clearance, and then use on-board information/tools to find 
a departure window, (e.g., safe separation from other approaching and departing traffic). 
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Many of the features of the GPS-T based SATS HVO concept are depicted in Fig. 1. Two IAFs, Cathy and 
Annie, are shown that also serve as the MAFIFs. There are also two DFs, Ginny and Ellen, located outside the SCA. 
SATS arrivals (Red and Blue aircraft) with alternating missed approaches, and departures (Green and Purple 
aircraft) are depicted in a “snapshot” in time: 

• The blue aircraft is able to enter the SCA at the IAF approach altitude upon receipt of the AMM message: “IAF, 

lateral entry’, follow none, missed approach Cathy”. Lateral entries are permitted when no other aircraft are 
assigned to the same fix and follow none indicates there is no traffic to follow. Aircraft coordinates descent and 
release from ATC communication to initiate the approach. The missed approach is depicted as a blue dashed 
path. 

• The red aircraft arrived by IFR 

clearance to the transition fix at 
4000ft and received an AMM 
message that says: “IAF, 

vertical entry, follow blue 
aircraft (identified by 
registration number), missed 
approach to Annie”. Vertical 
entries are required when there 
are other aircraft assigned to the 
fix or the approaching aircraft is 
too close for a lateral entry. 

• The purple aircraft is departing 
the SCA via the DF ELLEN. 

The pilot contacted ATC to get 
a clearance and release time and 
initiated the departure when 
safe separation from the other 
aircraft in the SCA was 
determined. 

• The green aircraft has already 

contacted ATC and gotten a 
clearance and release to depart 
through DF GINNY. The aircraft is holding short and using on-board tools to find an open slot in the arrival 
stream to take the active runway and depart. 

The piloted simulation experiment described in this paper was part of the validation and verification process of 
the SATS HVO concept that included multiple elements ranging from formal analysis of the procedures to a flight 
experiment, to full-system architecture prototype and public operational demonstration in June 2005. 



Figure 1. Schematic view of the SCA. 


ITT. Validation Experiment Description 
A. Objectives and Methodology 

In an effort to validate the concept from a pilot’s workload perspective, this simulation experiment sought to 
answer the questions “Can pilots safely and proficiently fly an airplane while performing SATS HVO procedures?” 
and “Do pilots perceive that workload, while performing HVO procedures, is no greater than flying in today’s 
system?”. Following the simulation experiment, the same research objectives and design guidelines were shared by a 
flight experiment conducted during the summer of 2004. Results from that activity are reported in a separate 
document [10]. 

During the simulation experiment, evaluation pilots (EPs) were asked to manually fly a general aviation (GA) 
aircraft simulator during IMC while performing a series of approach and departure scenarios. Two types of scenarios 
were used: 

1) Baseline scenarios were representative of current day IMC operations at non-radar equipped, non-towered 
airports (i.e., one operation at a time, managed by ATC). EPs were given departure and landing clearances as well as 
assigned altitudes and release instructions, and EPs were required to inform ATC of operation completion. 
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Traditional flight instrumentation and an experimental multi-function display (MFD) were incorporated into the 
simulated cockpit. All scenarios were designed for Melfa airport in Accomack County (MFV), Virginia. 

2) SATS HVO scenarios (Flereafter referred to as SATS scenarios) included only normal HVO operations and 
used the same starting location and traffic conditions as the Baseline scenarios. The same instrumentation and MFD 
were used during the SATS and Baseline scenarios. 

The simulation experiment measured the flight technical error, perceived workload, and situation awareness of 
EPs throughout the operations associated with both the SATS and Baseline scenarios. Workload and situation 
awareness data were collected at the end of each scenario, and usability questionnaires were administered after the 
completion of all runs. 

B. Evaluation Pilots 

Evaluation Pilots (EPs) were randomly drawn from a pool of pilots that met a set of criteria. The EPs included 
fifteen instrument rated, male subjects, 50% with commercial certificates and 50% with private pilot certificates. All 
EPs were IFR rated and current. All EPs were high performance capable. One pilot had received their certified flight 
instructor rating. All EPs held less than 1000 hours total flight time. EPs experience included 10.9 hours using the 
GPS as a primary navigation instrument. All EPs had experience in simulators and 93% had experience with a 
Florizontal Situation Indicator (HSI). 

C. Simulation Platform 

The simulation platform consisted of four pilot stations, a virtual traffic generator and an AMM with a plan view 
display. The pilot stations included a GA aircraft simulator, two flat panel displays for instrumentation, and one 
large plasma display for the out-the- 
window view. 

A yoke and trackball for flight 
control and instrument interaction and a 
set of head-phones/microphone allowed 
EPs to interact with other EPs and a 
simulated ATC station. All the pilot 
interface functionality designed for the 
SATS F1VO scenarios was displayed in 
the experimental MFD shown in Figure 
2. The MFD included a moving map 
with approach and path information, 
traffic and conflict detection and alerting 
symbology, procedure support 
functionality and an AMM communications window used to request and receive sequencing information. A virtual 
traffic generator provided virtual traffic operating in the SCA environment responding to AMM sequencing by 
means of a pilot model that performed F1VO flight procedures, responded to sequencing, and interacted with a 
simplified ATC model. The AMM provided sequencing information to all participating aircraft. A plan view display 
showed all the traffic in the SCA and vicinity (virtual traffic and pilot stations) and displayed the list of sequenced 
requesting aircraft. The ATC Model implemented required voice interactions for SCA transitions in SATS scenarios 
and approach and departure clearances in Baseline scenarios. 

The operational environment included single pilot scenarios where a single EP interacted with virtual traffic and 
multiple pilot scenarios where four EPs participated in the same common scenario. Single pilot scenarios were more 
predictable while traffic conditions in multiple pilot scenarios had an added degree of variability and fidelity. 

D. Experimental Design 

As described above, two types of procedures were tested to compare the EPs’ performance while completing 
both Baseline and SATS procedures. During SATS scenarios, EPs were required to follow SATS F1VO approach 
and departure procedures. The pilot aircraft interface included the FI VO experimental MFD with a moving map, 
approach and path information as well as AMM and procedure support information. During Baseline scenarios, EPs 
were required to follow today’s approach and departure procedures, one operation at a time, in a simulated ATC 
managed airspace. The pilot interface included the experimental MFD with a moving map and approach and path 
information (AMM and procedure support information were not provided). Five different types of scenarios were 
designed for each procedure. They included departures, single and multiple aircraft approaches and missed 



4 

American Institute of Aeronautics and Astronautics 





approaches. There were 10 scenarios (i.e., five scenarios associated with Baseline procedures, and five scenarios 
associated with SATS HVO procedures). 


Table 1: Experiment Design Matrix 




PROCEDURE TYPE 



Baseline 

SATS 


Departure 

EP,-15 

EP,-is 

SCENARIO 

Approach Without Traffic 

EPi., 5 

EP 1-15 

TYPE 

Approach With Virtual Traffic 

EP,.,5 

EP,.15 


Approach to Missed 
Approach with Virtual Traffic 

EP,.1S 

EPi-15 


Approach with Piloted Traffic 
(Linked Simulation) 

EPi-15 

EP,-15 


As shown in Table 1, the experiment design matrix used for data collection was a 2 (Procedure Type) x 5 
(Scenario Type), within-subject design in which the same 15 participants (i.e., low time instrument rated pilots) 
were assigned to each experimental cell (i.e., test condition). The two independent variables used in the experiment 
design were procedure type and scenario type, and EPs 1 through 15 (EP 1-15) were asked to perform all 10 test 
conditions twice in partially counterbalanced order. 

Dependent measures included EPs’ flight technical errors, subjective workload levels, as assessed using the 
Modified Cooper Harper (MCH) Rating Scale [13], and situation awareness levels as assessed using the Situational 
Awareness Rating Technique (SART) [11]. 

E. Experiment Scenarios 

Five different scenario types were designed that represented different approach and departure conditions. For 
each type two different procedures, Baseline and SATS were modeled. 

• Departure with approaching traffic The pilot task was to receive an ATC clearance, taxi onto the active 

runway, takeoff, and depart the airfield. In Baseline scenarios, ATC waited to give departure clearance to the 
EP’s aircraft until an approaching aircraft had landed and canceled its clearance. In HVO, the pilot was to self- 
determine when to depart by finding an opening in the approaching traffic flow. This was done by the pilot 
using a traffic display that showed aircraft on the approach and as long as the approaching aircraft was beyond 
the final approach fix (FAF), a departure was allowed. 

• Approach without traffic (no hofding required). The pilot task was to descend and fly the approach via ATC 

clearance during the Baseline scenarios. During the SATS scenarios, the pilot requested and received a “lateral 
entry, follow none” sequence from the AMM, received clearance to depart ATC space, and self-initiated the 
approach once in the SCA. Typically, the durations of the SATS scenarios were equivalent to those of the 
Baseline scenarios. 

• Approach with virtual traffic (holding required). This scenario clearly differentiated Baseline from HVO. In 

Baseline, the pilot waited behind two other aircraft in holding until they had landed and ATC provided 
clearance to begin the approach (i.e., 30+ minutes in holding). In HVO, the pilot followed the AMM sequencing 
behind the two other aircraft, self-separated from the other aircraft in the SCA, and self-initiated the approach 
by following advisories provided by a self-spacing software tool. The SATS scenarios’ durations were about 
half that of the Baseline scenarios. 

• Approach to missed approach with virtual traffic (holding required). This scenario included having the pilot 

fly the missed approach. SATS scenarios required flying the missed approach while self-separating in the SCA. 
The SATS scenarios’ durations were about two-thirds as long as the Baseline scenarios’ durations. 

• Multi-pilot linked simulation approaches (holding required). This scenario was flown with four EPs flying 

linked simulators to verify the HVO concept by introducing the variability of individually flown aircraft 
(instead of using one pilot in a scenario with precisely flown virtual traffic). The goal was to determine if pilots 
coidd conduct HVO while serving as traffic for one another. The time required to complete a multi-pilot HVO 
scenario was about one-quarter to one-third of the duration of a Baseline scenario. 
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IV. Analysis of Results 
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A. Flight Technical Error (FTE) 

FTE, defined as the difference between the position estimated by the aircraft navigation system and the desired 
aircraft position, is a well-established measure of pilot performance that can reflect the effects of multiple factors 
such as increased workload or reduced 
situation awareness. In this study, airspeed, 
altitude and lateral path deviation were 
measured during the experiment to assess the 
FTE of EPs flying Baseline and SATS 
scenarios. 

Airspeed, altitude and lateral path 
deviation were measured during the 
experiment to assess the FTE of pilots flying 
Baseline and SATS approach scenarios 
(departure scenarios were not assessed by 
FTE measures). Their task objective was to 
fly the scenarios while maintaining Instrument 
Rating Practical Test Standards (PTS) criteria 
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[14]. Data were analyzed by way of repeated measures Analysis of Variance (ANOVA) tests, with the main effect of 
procedure type being of primary interest, set a priori at a 5-percent significance level (a = .05) [15]. 

For every flight path parameter, the results obtained supported the experiment hypothesis that EPs’ flight 
performance during the SATS scenarios would be no worse than their flight performance during the Baseline 
scenarios. In some cases, results revealed that EPs’ flight performance during the SATS scenarios was significantly 
better than their flight performance during the Baseline scenarios. 

Airspeed: EPs were instructed to fly 

within the same airspeed envelope during the 
SATS and Baseline scenarios. The speed 
ranges for the initial, intermediate and final 
segments were 130 to 90 knots (kts), 110 to 
90 kts and 105 to 85 kts, respectively. As 
outlined below, three metrics were collected 
regarding airspeed. 

Number of excursions outside the airspeed 
envelope'. A statistically significant difference 
was found to exist between the total number 
of airspeed excursions associated with each 
procedure type as shown in Fig. 3. During the 
Baseline scenarios, 4.27 airspeed excursions 

occurred [Mean (M) = 4.27, Standard Deviation (SD) = 4.15, Sample Size (N) = 60]. During the SATS scenarios, 
3.18 airspeed excursions occurred (M = 3.18, SD = 1.81, N = 60). This finding indicates that fewer airspeed 
excursions occurred when EPs performed the SATS than when they performed the Baseline scenarios. 

Percentage of time within the airspeed 
envelope: As shown in Fig. 4, a statistically 
significant difference was found to exist 

between the percentages of time that EPs flew 
within the assigned airspeed envelope during 
each procedure type. During the Baseline 

scenarios, EPs flew within the defined 

airspeed envelope 77.78% of the time (M = 

77.78, SD = 19.27, N = 60). During the SATS 
scenarios, EPs flew within the defined 

airspeed envelope 89.87% of the time (M = 

89.87, SD = 7.43, N = 60). This finding 
indicates that EPs maintained airspeed within 
an assigned envelope more accurately when 


Figure 4. Percentage of time within airspeed envelope 



Figure 5. RMSE Airspeed with respect to target 
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they performed the SATS scenarios than when they performed the Baseline scenarios. 

RMSE with respect to airspeed target: A statistically significant difference was found to exist between the 
RMSE airspeed values associated with each procedure type as shown in Figure 7. A RMSE airspeed deviation of 
13.41 kts occurred during the Baseline scenarios (M = 13.41, SD = 7.65, N = 60), and a RMSE airspeed deviation of 
7.47 kts occurred during the SATS scenarios (M = 7.47, SD = 1.79, N = 60). This finding indicates that EPs 
maintained airspeed with respect to an assigned target value more accurately when they performed SATS scenarios 
than when they performed Baseline scenarios. 

Altitude: EPs were instructed to fly within the same altitude envelope for both Baseline and SATS scenarios, +/- 
100 ft of hold altitudes, -100 ft of “at or above” altitudes, and +100;-0 ft for minimum descent altitude (MDA) until 
missed approach point or visual transition to landing. This equates to: Initial Segment: Not below 1900 ft. 

Intermediate Segment: [Aircraft altitude at IF + 100 ft, 1400 ft]. Final Segment: [Aircraft altitude at FAF + 100 ft, 
MDA]. A statistically significant difference was not found to exist between the number of excursions outside the 
altitude envelope or between the percentage of time within the altitude envelope that occurred during the SATS 
scenarios as compared with the Baseline scenarios. 

Lateral path deviation: EPs were instructed to use target path as flight path for all segments using a target 
speed of 1 10 kts for turns during both the Baseline and the SATS scenarios. 

RMSE with respect to lateral path: A statistically significant difference was found to exist between the lateral 
path deviation RMSE values associated with 
each procedure type as shown in Fig. 6. A 
RMSE lateral path deviation of 0.12 mn 
occurred during the Baseline scenarios (M = 

0.12, SD = 0.19, N = 60), and a RMSE lateral 
path deviation of 0.05 nm occurred during the 
SATS scenarios (M = 0.05, SD = 0.03, N = 60). 

This finding indicates that EPs maintained 
lateral path deviation with respect to an 
assigned target value more accurately when 
they performed the SATS scenarios. 

B. Subjective Assessments of Workload 


EPs used the MCH Rating Scale to rate the 
level of workload that they experienced during 
each of the experiment’s 10 test conditions. 

Workload ratings could range on a scale from “1” (i.e., the instructed task was very easy/highly desirable; operator 
mental effort was minimal; and desired performance was easily attainable) to “10” (i.e., the instructed task was 
impossible; it could not be accomplished reliably). As reported below, nonparametric tests were employed as a 
conservative method for analyzing workload 
ratings associated with discrete rating scale items. 


EPs reported experiencing a mean workload 
rating of 1.69 when performing the SATS 
procedures (M = 1.69, SD = 0.54, N = 75) and a 
mean workload level of 2.59 when performing the 
Baseline procedures (M = 2.59, SD = 1.37, N = 

75) as shown in Fig. 7. A Wilcoxon Test (i.e., a 
nonparametric within-subject test appropriate for 
analyzing two related samples of ordinal data) 

[12] was performed on the mean workload ratings 
and revealed that EPs reported experiencing a 
lower level of workload when they performed the 
SATS procedures than when they performed the 
Baseline procedures (p = 0.05). This finding supports the hypothesis that pilots would not perceive an increase in 
workload when performing SATS procedures. 




Figure 6. RMSE Lateral Path Deviation with respect to 
Target Path 
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C. Subjective Assessments of Situation Awareness 

After each of the experiment’s 10 test conditions, all EPs completed a SART questionnaire that included the 
three dimensions of demand, supply, and understanding as well as two independent dimensions of traffic awareness 
and navigation guidance. For the dimensions of demand, supply, and understanding, EPs used a scale ranging from 1 
to 7 to report a score for each dimension. Global SART ratings were calculated by the formula: Situation Awareness 
= Understanding - (Demand - Supply) and could 
range from 1 (representing a low level of SA) to 14 
(representing a high level of SA). In the current study, 
calculated SART ratings ranged from 3 to 13. For 
traffic awareness and navigation guidance awareness, 
scores ranging from 2 to 7 on a scale of 1 (low) to 7 
(high) were collected from the EPs. As reported 
below, Wilcoxon Tests were used to analyze the 
SART, traffic awareness, and navigation guidance 
awareness scores. 

A mean SART rating of 9.6 was calculated for the 
SATS procedures (M = 9.6, SD = 1.97, N = 75), and a 
mean SART rating of 8.05 was calculated for the 
Baseline procedures (M = 8.05, SD = 2.68, N = 75) as 
shown in Fig. 8. The results of a Wilcoxon Test indicated that, at a statistically significant level, the SART ratings 
associated with the performance of the SATS procedures were higher than those associated with the performance of 
the Baseline procedures (p = 0.02). 

D. Subjective Assessment of Traffic Awareness 

With respect to traffic awareness, a mean rating of 

6.55 was calculated for the SATS procedures (M = 

6.55, SD = 0.72, N = 75), and a mean rating of 5.59 
was calculated for the Baseline procedures (M = 5.59, 

SD = 1.43, N = 75) as shown in Fig. 9. A Wilcoxon 
Test revealed that a statistically significant difference 
was found to exist between the mean traffic 
awareness rating for the SATS procedures and the 
mean traffic awareness rating for the Baseline 
procedures (p = 0.0003) as shown in Fig. 9. This 
finding indicates that EPs reported experiencing 
higher levels of traffic awareness when they 
performed the SATS procedures than when they 
performed the Baseline procedures. Therefore, the 
hypothesis that pilots would not experience a decrease in situation awareness when performing SATS procedures 
was supported. 
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Figure 8. Mean SART Rating 


E. Subjective Assessment of Navigation 
Guidance Awareness 

With respect to navigation guidance SA, a mean 
rating of 6.45 was calculated for the SATS 
procedures (M = 6.45, SD = 0.65, N = 75), and a 
mean rating of 5.44 was calculated for the Baseline 
procedures (M = 5.44, SD = 1.30, N = 75) as shown 
in Fig. 10. A Wilcoxon Test revealed that a 
statistically significant difference was found to exist 
between the mean navigation guidance awareness 
rating for the SATS procedures and the mean 
navigation guidance awareness rating for the 
Baseline procedures (p = 0.001). This finding 
indicates that EPs reported higher levels of 
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navigation guidance awareness when they performed the SATS procedures than when they performed the Baseline 
procedures. Again, the hypothesis that pilots would not experience a decrease in situation awareness when 
performing SATS procedures was supported. 

F. Usability Questionnaires 

EPs were asked a number of questions regarding the SATS HVO procedures and the experimental interface 
designed for the simulation experiment. Table 1 shows the procedure based questions and the summary of the 
responses given by the EPs. Overall, all of the EPs found the procedures easy to follow and not more demanding 
then current procedures. 


Table 1: Summary of Responses to Pilot Usability Questionnaires 



Question 

Summary’ of Responses 

1 

How easy was it to follow the SATS HVO flight 
procedures? 

100% of EPs responded “Easy” to “Very’ Easy” 
to follow. 

2 

If you found that the SATS HVO flight procedures were 
difficult to follow, was the difficultly associated with the 
procedures a result of: (turn into the form of a question) 

100% of EPs responded either no score or not 
applicable. 

3 

How complicated were the SATS HVO flight procedures 
compared to conventional flight procedures? 

All EPs rated the SATS HVO procedures as being 
between “Equally Complicated to” and 
“Significantly Less Complicated than” 

conventional flight procedures. 

4 

Was the holding procedure required within the SCA more 
difficult than the holding procedure required during 
conventional IFR? 

93% of EPs responded “No ” and 7%> responded 
“Somewhat” 

5 

How straightforward was the sequencing information 
provided during the SATS HVO scenarios? 

73 %> of EPs responded “Very’ Straight-Forward” 
and 27%> responded “ Straight-Forward ” 

6 

During the SAT HVO approaches, how useful was the 
"time to approach (TTA)" information? 

S3 % of EPs responded “ Very Useful” and 47% 
responded “Useful ” 


V. Summary 

This paper provides a description of a simulation experiment, that NASA Langley Research Center conducted as 
part of the SATS HVO concept validation process to determine pilot acceptability of normal HVO procedures. In 
addition, an in-depth description of the experiment’s results is provided that includes flight Technical Error (FTE) 
and subjective pilot ratings of workload and situation awareness. 

The comparative FTE analysis described includes lateral, vertical, and airspeed deviations from the Baseline and 
SATS HVO experimental flight procedures. The subjective pilot rating results are from MCH workload assessment 
ratings, SART ratings of situation awareness. Usability questionnaires were also given to pilots to assess their views 
on specific procedure issues. 

Based on FTE analysis, all evaluation subjects, low-time instrument-rated pilots, flew the HVO procedures 
safely and proficiently and with no more workload in comparison to today’s system (Baseline). Specifically, subject 
pilots maintained airspeed and lateral path more accurately when they performed the SATS scenarios than when 
they performed the Baseline scenarios. Subjects maintained altitude equally well in both SATS and Baseline 
scenarios. 

Pilots assessed their workload to be less with SATS HVO than Baseline scenarios, and situation awareness, 
traffic awareness, and navigation guidance awareness was greater with SATS than Baseline scenarios. 

The analysis of results presented in this paper indicate that for all the tested conditions, the answer to the 
questions “Can pilots safely and proficiently fly an airplane while performing SATS HVO procedures?” and “Do 
pilots perceive that workload, while performing HVO procedures, is no greater than flying in today’s system?” is 
unequivocally positive. The same research objectives and similar results were obtained by a flight test conducted by 
NASA in the summer of 2004. Other research activities followed this experiment that included piloted studies of 
non-normal procedures, Monte Carlo experiments and a full-system architecture test that was successfully shown to 
the public at the June 2005 SATS Technical Demonstration in Danville, VA. 
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